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The purpose of this study is to develop a framework for the accreditation of higher 
open and distance learning (ODL) programs in Turkey. The study was designed as a 
sequential monomethod multistrand mixed model including two strands which 
were both qualitative (QUAL^QUAL). In the first strand, both quantitative and 
qualitative data were collected through a three-round Delphi study with an expert 
panel consisting of 28 experts. In the second strand, qualitative data were collected 
via focus group interview. Based on a comprehensive literature review and the 
findings from the study, a framework was proposed including an initial 
accreditation process for new ODL programs and a re-accreditation process for 
ongoing programs. In addition, 35 criteria for new programs and 42 criteria for 
ongoing programs were developed to be used in the accreditation process. 

Keywords: Accreditation in open and distance learning, Delphi technique, focus 
group interview 

INTRODUCTION 


In recent years, quality assurance (QA) and accreditation in open and distance 
learning (ODL) have been among the top issues in the higher education agenda in 
the world (Higher Education Council [HEC], 2004). Chalmers and Johnston (2012, 
p.3) define accreditation as "a process of external quality review by an 
accreditation or certification body which enables an institution, program or course 
of study to be recognized or certified as meeting certain required standards." As it 
certifies the quality of an institution and/or its programs against a set of pre¬ 
determined standards, it is closely linked to quality assurance (Kilfoil, 2007). 

In each country, the purpose of establishing an accreditation system may vary 
according to the societal and cultural environments and national context (Jung et 
al., 2011; Stella, 2007). Whatever the purpose is, the primary goal of accreditation 
is to enhance quality and ensure quality assurance (Eaton, 2008). 

Due to international student mobility, emergence of cross-border universities and 
mobility of services in a globalized economy, accreditation practices have become 
one of the major issues in bilateral or multilateral relations of nations in the world 
(HEC, 2007). In parallel with these developments, fast growth in the number of 
ODL institutions and student enrollments have caused some concerns regarding the 
quality of programs delivered by these institutions (Belawati, 2010; Thorpe, 2003). 

97 



Daniel (2006) states that with the phenomenal proliferation of national and cross- 
border ODL across the world, quality matters more than ever. Yet, how to evaluate 
ODL within QA and accreditation practices is a controversial issue (Stella & Granam, 
2004; Welch & Glennie, 2005). Some researchers argue that quality of ODL should 
be judged by the same methodologies and criteria as face-to-face provision (Jung & 
Latchem, 2012; Stella & Granam, 2004). 

On the other hand, some researchers claim that ODL requires a different pedagogy 
and instructional design approach and many of the measures used in traditional 
accreditation reviews do not apply to ODL institutions or programmes; so, different 
measures and specific criteria should be used to evaluate ODL practices (Jung & 
Latchem, 2012; Kilfoil, 2007; Loane, 2001; Olcott, 2003; Stella & Granam, 2004; 
Swedish National Agency for Higher Education, 2008; Thorpe, 2003). Parallel to 
these arguments, QA and accreditation practices vary in each country. In most of 
the countries in the world, it can be observed that ODL is mostly accredited within 
the body of traditional accreditation agencies. In some of these agencies ODL is 
accredited in the same way as traditional education (For instance, agencies in 
Australia, Hong Kong, Taiwan, China, Thailand, Japan, Korea, Netherlands, Finland 
and Denmark) whereas in some other ones (For instance, agencies in Romania, 
Austria and Indonesia) it is accredited by using different methods and criteria (Jung 
& Latchem, 2007; Swedish National Agency for Higher Education, 2008). In addition 
to these, in some of the countries there are accrediting agencies which solely 
accredit ODL programs and institutions. For instance. Distance Education and 
Training Council (DETC) in the United States, Distance Education Council (DEC) in 
India and Open and Distance Learning Quality Council (ODL QC) in the United 
Kingdom are such kind of agencies (DEC Handbook, 2009; DETC, 2009; Kilfoil, 
2007). In sum, quality assurance and accreditation in ODL is still at an early stage 
of development (Jung, Wong, Baigaltugs, & Belawati, 2011). 

In the QA and accreditation process, different sets of standards or criteria can be 
used by the accreditation agencies. However, they concentrate on a number of 
common aspects and all educational QA guidelines stress the importance of policy 
and planning, human resources, programs, learning media, student support and 
student assessment (Belawati, 2010; Middlehurst, 2003). Although the criteria used 
for ODL in the accreditation process have common features with other types of 
learning, there are also differences (Belawati, 2010). Jung and Latchem (2007) 
state that ODL institutional QA procedures typically cover planning, management 
and administration, course design, development and delivery, learner support, 
assessment and technology applications and they often apply strict criteria to 
instructional design in particular. Hall (2003) indicates that guidelines feature three 
broad principles; these are curriculum, student support, assessment and evaluation. 

For instance, the guideline Best Practices for Electronically Offered Degree and 
Certificate Programs (2001) created by the 8 US regional accreditation commissions 
includes 5 components: Institutional context and commitment, curriculum and 
instruction, faculty support, student support, evaluation and assessment. 

For the E-xcellence label of European Association of Distance Teaching Universities 
(EADTU), 6 criteria were created (EADTU, 2009). These are strategic management, 
curriculum design, course design, course delivery, staff support and student 
support. 
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In the Quality Assurance Agency (QAA) in the UK, additional criteria are integrated 
into the existing criteria for flexible and distributed learning under the headings of 
delivery, support and assessment (QAA, 2004). Norwegian Association for Distance 
Education's (NADE) criteria are divided into prerequisites, implementation, results 
and follow-up. These phases are further divided into information and counseling, 
course development, education and organization (Swedish National Agency for 
Higher Education, 2008). 

In Turkey, not only in ODL but also in face-to-face education, implementation of a 
nation-wide QA and accreditation system is still a work in progress (Ozkul & 
Latchem, 2011). Currently, there is an approval mechanism for all the programs and 
the decision to start a program is made by the Higher Education Council (HEC) 
including ODL. A few program accreditation agencies have been established in the 
fields of engineering, psychology, science and literature for face-to-face education 
since 2005. In Turkey, currently more than 60 universities offer ODL programs 
(A.E. Ozkul, personal communication, September 23, 2011). Ozkul and Latchem 
(2011) indicate that in Turkey the emphasis has been on access, equity and 
capacity building rather than assuring quality in the products and processes in ODL. 
Latchem, Ozkul, Aydin, and Mutlu (2006) point out the necessity of quality 
assurance in Turkish ODL. 

Moreover, Turkey is one of the countries participating in the Bologna Process, 
which has an objective of creating a QA and accreditation system throughout 
Europe. So, it has been inevitable to establish an accreditation system in Turkish 
higher education. 

PURPOSE AND RESEARCH QUESTIONS 

The purpose of this study is to develop a framework for the accreditation of higher 
ODL programs within an accreditation system in Turkey. Accreditation is about both 
QA and quality improvement (Eaton, 2008). In the literature, it is observed that the 
terms accreditation and quality assurance are being used interchangeably. 

Thus, although the purpose is to develop a framework for the accreditation of ODL 
programs, also the term QA is used in some parts of the study. Research questions 
for this study are as follows: 

> How should be the appropriate accreditation process for new and ongoing 
ODL programs in Turkey? 

> Which standards/criteria should be used while evaluating new and 
ongoing ODL programs in the accreditation process? 

RESEARCH METHOD 

The study was designed as a sequential monomethod multistrand mixed model 
including two strands which were both qualitative (QUAL^QUAL). Creswell (2008, 
p.510) defines mixed method research design as "a procedure for collecting, 
analyzing and mixing both quantitative and qualitative data in a single study to 
understand a research problem." In sequential mixed designs, at least 2 strands 
occur chronologically and the strands are in relation with each other (Teddlie & 
Tashakkori, 2009). 


99 


In other words, the conclusions derived from the first strand constitute a basis for 
the formulation of design components for the next strand. 

First Strand 

In the first strand of the study, both quantitative and qualitative data were 
collected through a three-round Delphi study in order to identify the criteria that 
should be used in the accreditation process. Linstone & Turoff (2002, p.3) define 
Delphi as "a method for structuring a group communication process so that the 
process is effective in allowing a group of individuals, as a whole, to deal with a 
complex problem." Although the first applications of Delphi aimed to predict future, 
it changed in time and it began to be used in modified forms for various reasons 
such as planning, decision-making, problem-solving and evaluation (Delbecq, Van 
de Ven & Gustafson, 1975; Keeney, Hasson, & McKenna, 2001). 

It is a suitable method when there is incomplete knowledge about a problem or 
phenomenon (Garrod & Fyall, 2005; Hung, Altschuld, & Lee, 2008; Skulmoski, 
Hartman, & Krahn, 2007; Wiersma & Jurs, 2005). Keeney et al. (2001) state that 
Delphi method is based on applying a number of questionnaires iteratively; but 
unlike questionnaires, the Delphi aims to achieve consensus of opinion, judgment 
or choice. This is achieved through a series of rounds where statistical information 
about the given answers is fed back to participants using questionnaires. According 
to Rowe and Wright (1999), four key features may be regarded as necessary for 
defining a procedure as a Delphi. These are: anonymity of the participants, iteration 
of the questionnaires, controlled feedback, and the statistical aggregation of group 
response. Anonymity provides each participant to express his/her idea 
independently without being under group pressure (Hung et al., 2008; Westbrook, 
1997). 

With the iteration of the questionnaire over a number of rounds, participants have 
the opportunity to change their opinions and judgments (Hung et al., 2008; Rowe & 
Wright, 1999). Feedback mostly includes statistical information including 
aggregated group response (Mullen, 2003) and enables experts to review their 
responses in the context of the responses of other participants. 

In addition to these, Delphi enables participants, who are called as panelists, to 
participate asynchronously in the group communication process so that they can 
reflect on their answers (Delbecq et al., 1975; Keeney et al., 2001). It is a useful 
technique for collecting opinions from geographically dispersed experts who cannot 
meet face-to-face (Delbecq et al., 1975; Hung et al., 2008; Linstone & Turoff, 2002). 
Questionnaires can be sent to the participants via fax, post, e-mail (Gordon, 1994) 
or they can be applied through online questionnaire software programs. 

Accreditation of ODL programs is a relatively new issue in Turkey, so it was 
intended to benefit from the experiences of various academicians studying in the 
fields of ODL and/or accreditation. As they were geographically dispersed from 
each other and not able to meet face-to-face because of time and cost limitations, 
Delphi was considered to be the most suitable technique for collecting the data 
regarding the accreditation criteria in the study. Firstly, literature was reviewed in 
order to define the dimensions of the study and determine the characteristics of the 
participants. Based on the literature review, an open-ended question was prepared 
for the first round. Then, the expert panel was formed. 
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Selection of Panelists 

Delphi panelists are selected according to their subject matter expertise so that 
they can contribute to the topic (Hatcher & Colton, 2007). Since expert opinion is 
sought, a purposive sampling method is used in Delphi (Sahin, 2010; Skulmoski et 
al., 2007; Franklin & Hart, 2006). Also snowball sampling is commonly used 
(Skulmoski et al., 2007). There is no agreement regarding the size of the panel and 
in the Delphi literature it is indicated that panel size varies from a few to hundreds 
of experts (Grisham, 2008; Hatcher & Colton, 2007; Sahin, 2001; Skulmoski et al., 
2007; Wiersma & Jurs, 2005; Williams 8i Webb, 1994). According to Delbecq et al. 
(1975), with a homogenous group of people, 10 to 15 participants might be 
enough. They indicate that few new ideas are generated within a homogenous 
group once the size exceeds 30 well-chosen participants. In this regard, in the 
study, Delphi panel consisted of 28 Turkish academicians who were experts at 
accreditation/quality assurance and/or ODL from 17 different universities in 
Turkey. They were selected through purposive and snowball sampling method. 

Delphi Process 

Three rounds occurred in the Delphi study. In Delphis, rounds are repeated until the 
consensus is achieved; however, three rounds of iterations are commonly viewed as 
sufficient for consensus (Delbecg et al., 1975; Hung, Altschuld & Lee, 2008; 
Linstone & Turoff, 2002). Thus, the study was stopped on the third round. The 
whole Delphi process was completed approximately in 9 months. 

First Round 

First round questionnaire usually consists of open-ended questions (Delbecq et al., 
1975; Franklin & Hart, 2007). Therefore, a questionnaire composing of an open- 
ended question was sent to the panelists via e-mail and 16 experts responded to 
the questions in the first round. Responses given to the open-ended questions were 
analyzed qualitatively by grouping and transforming them into question forms that 
could be asked as quality cirteria in the accreditation process under six main 
elements: Management, Organization and Planning, Program and Course Design, 
Learners and Support Services, Staff and Support Services, Assessment and 
Learning Outcomes, and Evaluation. In order to provide reliability, another 
researcher coded the data and intercoder reliability was calculated as 93 % with 
the formulation suggested by Miles and Huberman (1994). 

Second Round 

The second and subsequent rounds consist of the questionnaires that usually 
include quantitative rating or ranking techniques and require quantitative analysis 
(Powell, 2003). For the second round, after evaluating the responses given in the 
first round, a questionnaire was developed consisting of 95 accreditation criteria 
questions in which the experts were asked to mark one of the statements Should be 
definitely asked (4), Should be asked (3), Might be possible if not asked (2), Not 
necessarily should be asked (1) in a 4-point Likert type scale. 

In addition, by adding a comment section next to each item in the questionnaire, 
the experts were asked to make comments regarding the items in cases where they 
found it necessary. The questionnaire was sent online via LimeSurvey, which is an 
online questionnaire software. Consequently, 22 experts responded to the 
questionnaire in the second round. 
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Data analysis method used in the Delphi technique may change according to the 
purpose of the research, structure of the rounds, types of research questions and 
number of participants, and consensus can be defined in a variety of ways (Powell, 
2003). In most Delphis, consensus is achieved when a certain percentage of the 
given responses to the items fall within a prescribed range (Scheibe, Skutsch & 
Schofer, 2002). Determination of consensus level depends on the topic of the 
research; for instance in a topic related with health, having 100% consensus might 
be required (Keeney, Hasson, & McKenna, 2006). However, Williams and Webb 
(1994) state that some researchers accepted the consensus level as 55% in the 
studies they conducted. In addition, as for consensus criteria in Delphi studies, 
measures of central tendency (e.g. mean, mode, median) and dispersion (e.g. 
standard deviation, interquartile range) are used. Mitchell (1991) asserts that the 
median is a robust estimator of location because it is not strongly influenced by 
outlying data points whereas the mean is very sensitive to data in the tails of a 
distribution. 

Similarly, according to Gordon (1994), the group judgment should be based on the 
median rather than the mean, since single extreme answers can pull the mean 
unrealistically. Therefore, for the evaluation of the second round responses, 
frequencies, median and interquartile range (IQR) were calculated by using SPSS 
software to represent group opinion. In the analysis, the median shows the level of 
agreement at which half of the responses fall above and half fall below and the IQR 
is the absolute value of the difference between the 25 th and the 75 th quartiles 
(Sahin, 2010). If the IQR is low, it means that the panel is in agreement. 

Table: 1 

Definition of Consensus 


Definition of Consensus 

2nd Round median > 3, IQR < 1, frequency 3-4 > %80 

3rd Round median > 5, IQR < 1, frequency 5-7 > %90 


In the study, the first criterion used to indicate consensus was a level of 80% which 
showed that more than %80 respondents rated 3 or 4 on a 4-point scale. The 
second criterion was a median of 3 or above and an IQR of 1 or below. In other 
words, items with a frequency of minimum 80%, with a median of minimum 3, with 
an IQR of maximum 1 were considered to show consensus as shown in Table 1. As a 
result of the evaluation, 21 items out of 95 remained below the determined 
consensus level and omitted. 

Third Round 

A questionnaire was prepared composing of the remaining 74 items for the third 
round. In the questionnaire, the panelists were asked to mark the importance 
degree of the 74 accreditation criteria questions that could be asked in the 
accreditation process in a 7-point scale, 7 indicating a criterion very important 
Statistical information of the second round (the frequencies, the median and the 
IQR of each item) was given as feedback to the panelists. Also a comment section 
was added next to each item. The questionnaire was sent to the panelists via 
LimeSurvey. Consequently, 18 experts responded to the third round questionnaire. 
The responses of the third round were evaluated according to the determined 
consensus criteria by using SPSS. 
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For the third round, the criteria used to indicate consensus was a frequency of 
90%, a median of 5 or above and an IQR of 1 or below in a 7-point scale as shown 
in Table 1. As a result, 8 items were omitted as they remained below the consensus 
level. 

Second Strand 

The goal of the second strand was to determine the appropriate accreditation 
process for ODL programs in Turkey and to discuss in-depth the results of the first 
strand regarding the accreditation criteria. In order to achieve this goal, data were 
collected via focus group interview, which is a process of collecting data through 
interviews with a group of people on a specific topic (Creswell, 2008; Patton, 
2002). The researcher asks a few questions on the topic and gets responses from all 
participants in the group (Creswell, 2008). Focus group interviews are cost- 
effective ways of collecting in-depth information in a relatively short period of time 
(Johnson & Christensen, 2008). In this study, as the accreditation of ODL is a novel 
issue in Turkey, data were considered to be collected via focus group interview best 
in order to get various kinds of views from the experts in an interactive 
environment. 

Selection of Participants 

Participants of focus group interview were 21 academicians, who were experts in 
accreditation/quality assurance and/or ODL from 12 different universities in Turkey 
and Turkish Republic of Northern Cyprus. They were selected through purposive 
and snowball sampling method. 

Firstly, several academicians who met the expertise criteria working in various 
universities were contacted as well as the Delphi panel experts via e-mail or phone. 
Consequently, 21 academicians, 11 of whom were the Delphi panelists, accepted to 
participate in the focus group interview. 

Focus Group Interview Process 

The preparation of the focus group interview started on January, 2011 and the 
interview was conducted on February 26, 2011 in Eskisehir. A couple of days before 
the study, the participants were informed about the interview. They were given 
detailed information about the purpose of the study, the schedule and the names of 
the other participants via e-mail. In addition to this, participants were given more 
detailed information on the day of the interview, and a list containing the 
accreditation criteria developed in the first strand was distributed to the 
participants. 

At the beginning of the interview, it was intended to conduct 3 different groups of 
interviews, each group consisting of 7 people, in order to get a variety of 
perspectives. However, the participants indicated that discussing the topic all 
together would be more effective than having 3 different groups as the 
accreditation is a very novel issue in Turkey. Due to the preference of the 
participants, instead of creating small groups, a single focus group interview was 
conducted including 21 people. The focus group interview was arranged through 
one day, between 09:00 and 15:30 in three sessions each of which lasted for one 
and a half hours. 
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This can be regarded as a limitation of this study because in the literature it is 
indicated that focus group interviews consist of 4 to 6 or 6 tolO people and they are 


conducted for one to two hours (Creswell, 2008; Patton, 2002). In a focus group 
interview, a moderator leads the discussion and he/she must know how to facilitate 
group discussion (Johnson & Christensen, 2008). The moderator in this study was 
one of the researchers of this study who met the expertise criteria used in the 
participant selection of the Delphi and focus group interview. He had good 
interpersonal skills and experience in managing group communication processes. 
The focus group interview was recorded using an audiotape so that the data could 
be analyzed later. In addition, the other researcher took notes during the sessions. 
As Creswell (2008) suggested, after transcribing the records of each session, the 
researchers read the data several times in order to get a general sense of the 
material, and coded them. They were coded according to the questions posed in the 
focus group interview. Then, the data were explained and interpreted. In order to 
provide reliability, another researcher coded the data and intercoder reliability was 
calculated as 85% with the formulation suggested by Miles and Huberman (1994). 

Trustworthiness of the Study 

Traditional approaches used for providing validity and reliability are not easily 
applicable to the Delphi studies (Fish 8i Busby, 2005). However, several different 
techniques are used to determine whether Delphi studies are valid and reliable or 
not. For instance, in Delphi studies, content validity is sought, which is usually 
identified according to the related literature and expert judgment (Paykoc & Ok, 
1990). In order to provide content validity in this study, the researchers carried out 
an extensive literature review and benefited from the views of some other experts 
throughout the research. In addition, as the content in Delphi studies is created by 
the expert panel, the validity is directly related to the selection of the panel experts 
(Fish & Busby, 2005). So, it is very important to define clearly the qualifications 
that the panel members should have and to select the members according to those 
determined qualifications (Clayton, 1997). Therefore, in the study, the required 
qualifications were defined clearly, and the experts were selected among the ones 
having those qualifications. Besides, for the verification of results, Skulmoski et al. 
(2007) suggest conducting a follow-up study after Delphi, such as interviews or 
survey. In this study, the results of the Delphi were discussed in-depth in the focus 
group interview. 

In the literature, it is accepted that the Delphi technique is as reliable as the other 
techniqes for forecasting, creating consensus of opinion, making decision, etc. 
(Paykoc & Ok, 1990; Ono & Wedemeyer, 1994). According to Fish and Busby (2005, 
p.250), "reliability between first and second questionnaires can be estimated by 
exploring the consensus rates of the respondents and if a reasonable level of 
consensus is produced on many items on the second questionnaire, it is likely that a 
researcher has adequately summarized the meaning behind responses of the first 
questionnaire." In this study, consensus was achieved on many items both in the 
second and third round. 

Moreover, Mitchell (1991) states that clarity of the items affects reliability of the 
results and suggests testing the questionnaires in advance. Therefore, in order to 
ensure the clarity of items, the questionnaires were checked by the experts in the 
field of assessment and evaluation and Turkish language. Besides, the 
questionnaires were applied to 3 people who have similar backgrounds with the 
panelists and revised before each round. 
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RESULTS 


The Accreditation Process 

Based on the findings, literature and the quality concerns about the Turkish ODL 
programs, a different accreditation process is proposed apart from other country 
cases in order to increase the quality of rapidly increasing ODL programs and to 
improve these programs continuously. It is proposed to establish a separate, self- 
governing accreditation agency responsible for quality assurance and accreditation 
activities of ODL, and this agency is expected work in cooperation with the field 
specific program accreditation agencies that had been established or will be 
established in the Bologna Process. The ODL accreditation agency is thought not 
only to accredit the programs or institutions, but also to give a counseling service 
for higher education institutions, by leading and supporting the research towards 
increasing the quality of ODL at the same time. In this framework, giving an initial 
accreditation to new opening programs and afterwards repeating the accreditation 
process in every 5 years is suggested. The initial accreditation process might be 
thought as the permission to open a program. 

When the accreditation processes of other countries are examined, it can be seen 
that while in some countries the process for new and ongoing programs is the 
same, in some others a different process is followed for the newly opened 
programs. 

For instance, in the accreditation agency NVAO in the Netherlands, different 
regulations are being used for the new and ongoing programs (Frederiks, 2010). In 
FH Council, which is one of the accreditation agencies that operates in Austria, 
initial accreditation is given to new programs; and afterwards the accreditation is 
being renewed once in every 5 years (Sohm, 2004), as similar to the suggestions of 
the experts participated in this research. 

In this study, for giving an initial accreditation to a newly opening ODL program, 
the stages are suggested to occur as given below: 

> The higher education institution applies to the Higher Education Council 
(HEC) to open an ODL program. 

> The application is examined by HEC and if HEC approves the application to 
be evaluated, then forwards it to the field specific program accreditation 
agency for an accreditation review. 

> The higher education institution prepares a self-assessment report and 
presents it to the program accreditation agency. 

> The program accreditation agency contacts with the ODL accreditation 
agency and these two agencies examine the self-assessment report 
together within a committee that will be established collectively. 

> The two agencies organize a team visit consisting of 3 people; 2 field 
experts from the field specific program accreditation agency and 1 ODL 
expert from the ODL accreditation agency. 

> The team prepares an evaluation report and presents it to the committee 
that had been established collectively before in Stage 4. 

> The committee determines its opinion regarding whether the program can 
be opened or not and notifies it to HEC. 

> HEC decides whether to open the program or not. 
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The initial accreditation process is estimated to be completed minimum in 2 
months. In the suggested model, after the initial accreditation, renewing the 
accreditation of ongoing programs once in every 5 years is being suggested. 

In the process of renewing the accreditation of an ODL program in which an initial 
accreditation is given, the stages are suggested to occur as given below: 

> The Higher Education Institution Applies To The Field Specific Program 
Accreditation Agency, Prepares A Self-Assessment Report And Presents It. 

> The Program Accreditation Agency Contacts With The ODL Accreditation 
Agency And These Two Agencies Examine The Self-Assessment Report 
Together Within A Committee That Will Be Established Collectively. 

> The Two Agencies Organize A Team Visit Consisting Of 4 People; 1 Field 
Expert From The Field Specific Program Accreditation Agency, 2 ODL 
Experts From The ODL Accreditation Agency-1 From Technical Area And 
The Other From Design, Management, Etc. Areas- And 1 ODL Student. 

> The Team Prepares An Evaluation Report And Presents It To The 
Committee That Had Been Established Collectively Before In Stage 2. 

> The Committee Decides Whether To Renew The Accreditation Or Not, And 
Makes Comments And Suggestions To The Institution In Order To Improve 
The Program. 

Duration of a re-accreditation process may change according to the higher 
education institution's self-assessment report preparation time; however the whole 
process is estimated to be completed at least in 3 months. The proposed framework 
for the accreditation of new and ongoing ODL programs is shown in Figure 1. 
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Figure: 1 

Proposed Accreditation Framework for ODL Programs 
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During both initial accreditation and re-accreditation process, the field specific 
program accreditation agency is supposed to focus on whether criteria established 
towards the learning outcomes are carried out or not, while the ODL accreditation 
agency is inspecting whether the ODL methods are applied effectively or not, by 
evaluating the practice in terms of the method. 

In addition to this, ODL programs which already operate and had not been in the 
accreditation process before have been suggested to go through the same 
accreditation process as a pre-accredited program's accreditation renewal process. 

The Criteria That Will Be Used In The Accreditation Process 

In this study, based on the findings and literature, 35 criteria for initial 
accreditation of newly opening programs; and 42 criteria for re-accreditation of 
ongoing programs have been developed. 
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Figure 2 

Dimensions of the Criteria Matrix 

Differently from the criteria used in other countries, a 2-dimensional matrix has 
been created on the developed criteria. As shown in Figure 2, the first dimension of 
the matrix consists of 3 levels; the institution, the program and the courses. In each 
of the levels, there exist criteria regarding the organizational structure, the design 
and production, the support services, the assessment and evaluation elements that 
constitute the second dimension of the matrix. The elements that exist in the 
second dimension of the matrix are the most featured elements of ODL. 
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Table: 2 

Initial Accreditation Criteria Matrix 


Organizational Design and _ Assessment and 

Structure production pp evaluation 

Institution 
Program 

Courses 


I, 2, 3, 4, 6,7, 8 

II, 12, 13,14, 
15, 19, 24 

30, 31 


16, 25, 
26 
35 


5, 10 

17, 20, 21, 
22, 23 
28, 33, 34 


9 

18, 27 
29, 32 


Table: 3 

Re-accreditation Criteria Matrix 


Organizational Design and 

Structure production 


Support 


Assessment and 
evaluation 


Institution 1, 2, 3, 4, 6, 7, 8 

Program 11,12, 13,14, 

15,19, 24 

Courses 33, 34, 35 


5, 10 

16, 25, 26 17, 20, 21, 

22, 23 

39, 42 32, 38, 40 


9 

18, 27, 28, 
29, 30, 31 

36, 37, 41 


The suggested criteria for the newly opening and ongoing programs are given in 
Appendix A and Appendix B respectively; the relation of suggested criteria and their 
related titles are given in Table 2 and Table 3 in the criteria matrix by indicating the 
criteria number. 

CONCLUSION 

QA and accreditation practices in ODL are still a relatively new issue compared with 
face-to-face education in the world (Jung et al v 2011). Similarly, QA and 
accreditation processes are still at an early stage of development in Turkey (Ozkul 
& Latchem, 2011). As Jung et al. (2011) indicate that these practices are affected 
by societal and cultural environments of each country; it was intended to develop a 
framework for the accreditation of ODL programs that meets Turkey's social, 
cultural and educational requirements by seeking expert opinions in this study. As a 
result, a different accreditation process is proposed when compared to the practices 
of other countries. The major difference from other country cases is the provision of 
cooperation between the ODL accreditation agency and the field specific program 
accreditation agencies. Moreover, two separate processes have been proposed for 
new and ongoing ODL programs. Besides, 35 criteria for the newly opening 
programs; and 42 criteria for ongoing programs have been developed to be used in 
the proposed accreditation process. Differently from the criteria used in other 
countries, a 2-dimensional matrix has been created on the developed criteria. In 
the next step, it is intended to create a rubric from these criteria and apply it to a 
few ODL programs in a pilot study. In Turkey, as the accreditation of ODL is a novel 
issue, more research is needed including a wide range of stakeholders such as 
administrators working in the ODL institutions, employers or students as well as 
academicians. 
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In Turkey, it is crucial to ensure the autonomy of accreditation agencies to have a 
robust ODL accreditation system. In other words, accreditation decisions should be 
made independent from politics. Especially developing countries confront problems 
in establishing effective accreditation sysytems since they do not have qualified 
human resources in the field of accreditation and they lack sufficient financial 
resources. In order to have an effective system, a quality culture should be adopted 
in the educational institutions by including all the staff in the accreditation process. 
Moreover, the staff should be trained about the philosophy, purpose and process of 
an accreditation system. 

Finally, the ODL accreditation criteria require being dynamic because of the 
continuously changing features of ODL due to the developments in information and 
communication technologies. Therefore, they should be updated according to the 
requirements of the ODL field by the accreditation agencies. Besides, the 
qualifications determined in the National Qualifications Framework can be reviewed 
in terms of ODL programs. 
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APPENDIX A 

INITIAL ACCREDITATION CRITERIA 
Institution 


1. H 
as the institution notified its vision, mission, plans, objectives and goals clearly 
regarding ODL? 

2. H 
as the institution constructed a management and an organizational structure for 
carrying out ODL practices? 

3. A 
re there enough and competent human resources in the institution to carry out 
plans towards ODL? 

4. D 
oes the institution have the systems that will make ODL system work efficiently 
for the ODL personnel and students, such as the support services, learning 
management system, student information system, decision support system, 
information management system, library facilities, staff management system, 
etc.? 

5. D 
oes the institution have systems providing ODL students to reach the library 
facilities and sources easily? 

6. D 
oes the institution have sufficient technical infrastructure (eg. the server 
services, enough capacity to access the internet, etc.) to run ODL programs? 

7. H 
ave the precautions been taken regarding the security of the technical 
infrastructure in the institution? 

8. H5d 

oes the institution have policies towards disabled students? 

9. A 
re there any evaluation systems and development plans in which continuous 
feedback can be taken towards the improvement of the ODL system in the 
institution? 



10 . 


oes the institution have plans which will coach the administrative and academic 
personnel who will take place in running the ODL system and contribute to their 
professional development? 


Program 

11. I 

s the information below notified clearly? 

Type of the program (professional or academic) 

Objectives and aims of the program 

Outcomes of the program (Compatible program qualifications with the National 
Qualifications Framework and related field of study qualifications) 

Educational goals 
Educational opportunities 

The state of the program; whether it is a discipline or inter-disciplinary (the 
relationship of the program with other disciplines) 

Entry requirements 
Graduation requirements 
Job opportunities 

Flexible teaching and assessment strategies that fits ODL 


12. I 
s there adequate number of academic and administrative staff to run the 
program such as faculty, support personnel, technical infrastructure experts, 
assessment experts, etc.? 

13. I 
s the program compatible with the needs and objectives of the country/region 
and sector? 

14. I 
s the program compatible with the institutional vision, mission, objectives, aims 
and policies? 

15. A 
re there any financial regulations and plans (on topics such as copyright, 
salaries, fees, costs, etc.) to start and sustain the program? 

16. I 
s it intended to have a universal instructional design in the program that is 
compatible with the institution's policies towards disabled students? 

17. I 
s it intended to ensure learning opportunities for students other than courses 
such as library facilities, learning sources, activities, etc. in the program? 

18. 1 
s it intended to construct evaluation systems and improvement plans to have 
continuous feedback towards developing the program? 

19. I 
s it intended to establish an internal quality assurance process in the program 
related with design, production and assessment of the course materials? 

20 . 1 
s it intended to provide administrative support for students in variously 
environments, such as registration, examination dates, documents necessary 
for military issues, etc.? 

21. I 

s it intended to provide continuous and immediate technological support for 


students in various environments, such as password issues, access to courses, 
technical problems, etc.? 

22. I 
s it intended to provide support for academic and administrative staff who take 
place in the program while they are implementing the program? 

23. I 
s it intended to construct any systems that enable students to interact socially 
other than courses? 

24. H 
ave the competencies, job descriptions, roles, duty and responsibilities of full¬ 
time and part-time staff (both academic and administrative) been identified 
clearly? 

25. H 
ave the content, instructional methods and media used in the program been 
designed to enable students to gain the required program qualifications? 

26. H 
as the program been designed to provide students the opportunity to plan their 
educational process (course selection, the period to finish the program, etc.) by 
themselves? 

27. H 
ave the assessment methods and tools been designed to provide a valid and 
reliable measurement of the program qualifications? 


Course 

28. I 
s it intended to provide support services for the academic staff while developing 
the courses? 

29. H 
ave the objectives, aims, learning outcomes, instructional methods, content and 
the assessment methods of the courses been identified clearly? 

30. H 
ave the courses been determined according to the credit system of the 
university and ECTS? 

31. A 
re the learning outcomes of the courses compatible with the program 
qualifications? (Are there any program qualifications and learning outcomes 
matrix?) 

32. F 
or each course, is it explained how to ensure valid and reliable assessment 
methods and tools proper for the program's assessment strategy? 

33. I 
s it intended to provide educational support for students whenever they need 
regarding the course content? 

34. I 
s it intended to provide any information sources for students in the courses to 
improve themselves other than the learning outcomes? 

35. 117i 
s it intended to provide adequate student-student, student-teacher, student- 
content and student-interface interaction in the courses that will provide to 
reach the learning outcomes? 


APPENDIX B 


RE-ACCREDITATION CRITERIA 


Institution 

1. H 
as the institution notified its vision, mission, plans, objectives and goals clearly 
regarding ODL? 

2. H 
as the institution constructed a management and an organizational structure for 
carrying out ODL practices? 

3. A 
re there enough and competent human resources in the institution to carry out 
plans towards ODL? 

4. H 
as the institution constructed the systems that will make ODL system work 
efficiently for the ODL personnel and students, such as the support services, 
learning management system, student information system, decision support 
system, information management system, library facilities, staff management 
system, etc.? 


5. D 
oes the institution have systems providing ODL students to reach the library 
facilities and sources easily? 

6. D 
oes the institution have sufficient technical infrastructure (eg. the server 
services, enough capacity to access the internet, etc.) to run ODL programs? 

7. H 
ave the precautions been taken regarding the security of the technical 
infrastructure in the institution? 

8. D 
oes the institution have policies towards disabled students? 

9. A 
re there any evaluation systems and development plans in which continuous 
feedback can be taken towards the improvement of the ODL system in the 
institution? 

10. D 
oes the institution have plans which will coach the administrative and academic 
personnel who take place in running the ODL system and contribute to their 
professional development? 


Program 

11. I 

s the information below notified clearly? 

Type of the program (professional or academic) 

Objectives and aims of the program 

Outcomes of the program (Compatible program qualifications with the Nationally 
Qualifications Framework and related field of study qualifications) 

Educational goals 
Educational opportunities 

The state of the program; whether it is a discipline or inter-disciplinary (the 
relationship of the program with other disciplines) 


Entry requirements 
Graduation requirements 
Job opportunities 

Flexible teaching and assessment strategies that fits ODL 


12. I 
s there adequate number of academic and administrative staff to run the 
program such as faculty, support personnel, technical infrastructure experts, 
assessment experts, etc.? 

13. I 
s the program compatible with the needs and objectives of the country/region 
and sector? 

14. 1 
s the program compatible with the institutional vision, mission, objectives, aims 
and policies? 

15. A 
re there any financial regulations and plans (on topics such as copyright, 
salaries, fees, costs, etc.) to have a sustainable program? 

16. I 
s there a universal instructional design in the program that is compatible with 
the institution's policies towards disabled students? 

17. I 
n the program, are there any learning opportunities for students other than 
courses such as library facilities, learning sources, activities, etc.? 

18. A 
re there any evaluation systems and improvement plans to have continuous 
feedback towards developing the program? 

19. I 
s there an internal quality assurance process in the program related with 
design, production and evaluation of the course materials? 

20. H 
ave the students been provided administrative support in various environments, 
such as registration, examination dates, documents necessary for military 
issues, etc.? 

21. H 
ave the students been provided continuous and immediate technological 
support in various environments, such as password issues, access to courses, 
technical problems, etc.? 

22. D 
oes the academic and administrative staff who takes place in the program have 
support while they are implementing the program? 

23. A 
re there any systems that enable students to interact socially other than 
courses? 

24. H 
ave the competencies, job descriptions, roles, duty and responsibilities of full¬ 
time and part-time staff (both academic and administrative) been identified 
clearly? 

25. H 
ave the content, instructional methods and media used in the program been 
designed to enable students to gain the required program qualifications? 

26. H 
as the program been designed to provide students the opportunity to planll9 
their educational process (course selection, the period to finish the program, 
etc.) by themselves? 

27. D 
o assessment methods and tools provide a valid and reliable measurement of 
the program qualifications? 



28. H 
ave the program qualifications expected from the graduated students and 
realized program qualifications been parallel? 

29. H 
as any feedback been gotten from the students and the academic staff 
regarding the program? 

30. H 
ave the student satisfaction rates been measured and are there any 
improvements made according to the results? 

31. I 
s there any statistical information related with the program? Are there any 
policy decisions taken according to the evaluation results of this statistical 
information? 

Course 

32. A 
re there any support services for the academic staff while developing the 
courses? 

33. A 
re the objectives, aims, learning outcomes, instructional methods, content and 
the assessment methods of the courses presented clearly to the students in 
various environments? 

34. H 
ave the courses been determined according to the credit system of the 
university and ECTS? 

35. A 
re there clear explanations presented to students regarding the courses 
(methods used, media, environments, tools, instructions, messages, etc.)? 

36. A 
re the learning outcomes of the courses compatible with the program 
qualifications? (Are there any program qualifications and learning outcomes 
matrix?) 

37. F 
or each course, are there valid and reliable assessment methods and tools 
proper for the program's assessment strategy? 

38. 1 
s there an educational support service whenever the students need regarding 
the course content? 

39. A 
re the learning materials used in the courses appropriate in terms of usibility 
(visual design, easy access, navigation)? 

40. A 
re there any information sources in the courses provided for the students to 
improve themselves other than the learning outcomes? 

41. D 
o teaching-learning activities provide students to reach the expected learning 
outcomes? 

42. D 
oes the design of the courses provide adequate student-student, student- 
teacher, student-content and student-interface interaction that will provide to 
reach the learning outcomes? 
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